Entity Duplicates

Each Entity Type has a number of properties that can be used to help both identify and describe it's attributes. For example a vehicle Entity type may have properties for - licence plate, make, model and colour and a person type might include first, middle and last name, date-of-birth, known as, ID, role, appearance etc.

When a duplicate check value is assignedthis is managed in the Settings to any of the properties (either as a single property or as a combination of properties), where the value \ details of those properties match an existing Entity (of the same Type) and equals or exceeds the Duplicate Threshold value, this triggers the identification of a potential duplicate.

The duplicate values assigned to individual Entity properties is managed within the Settings by your Organisation as is the Threshold value.
The duplicate threshold value is the number used that when equaled or exceeded (by the sum of the properties with duplicate values) triggers the identification of a potential duplicate.

Here are some examplesThese may differ from your Organisation's Entity types and properties for different Entity Types and their properties and shows where a duplicate value has been assigned to a property.
The duplicate threshold value that must be equaled or exceeded in this example is 7.

Entity Type - Vehicle	Duplicate value	Entity Type - Person	Duplicate value	Entity Type - Organisation	Duplicate value
Properties		Properties		Properties
Make		Surname	3	Name	3
Model		First Name	2	Category	2
Colour		Middle Name		Location \ Branch
Licence Plate	7	Date of Birth	2	Company Number	4
		ID Employee ID, Prisoner ID, Client ID. E.G any identifier that provides a unique ID	7
		Known As

NOTE: Remember that the Duplicate property values (including the threshold) used in these examples may be different to the Duplicate values set by your Organisation. If unsure of the duplicate settings it would be useful to request (from your ModalConnect system administrator), a list of any Entity Types and those properties that have (and include) a duplicate value.

Vehicle Entity Types are identified as potential duplicates where the Registration number property detail matches (as the Duplicate value of that property meets the threshold). Duplicate value = 7

Person Entity Types where properties with a Duplicate value match, either to a single property or multiple properties (with a duplicate value assigned) match and the value or combined value equals or exceeds the Duplicate threshold of 7

In this example the 'Person' Entity property details match for First Name (duplicate value 2), Surname (duplicate value 3) and Date of Birth (duplicate value 2) and the combination of the duplicate values of these properties meet the Duplicate threshold of 7 so the Entity is identified as a Potential Duplicate. 2 + 3 + 2 = 7

In the following example the 'Person' Entity is identified as a Potential Duplicate due a match of the unique ID which has a duplicate value of 7 which equals \ meets the threshold.
So even though not all of the other properties match, the ID property which is supposed to be a unique ID matches.

In the following example for an Entity Type of 'Organisation' - a Potential Duplicate has been identified as the Name property (duplicate value of 3) and the Company number property (duplicate value of 4) match so with a combined value of 7 it meets the threshold for Organisation types and triggers the Potential duplicate. 3+4 = 7

How Duplicates can occur

Duplicate Entities can occur when the same Entity is created more than once either in different Documents or even within the same Document, or perhaps simply due to a data entry error E.g. the same ID number given to more than 1 Entity (of the same type).

If potential Entity duplication has been detected within different Documents from the same File or Documents from other FilesPermissions apply for the visibilty of the File within the same Department or even across other Departments, these are identified and can be previewed in the associated File Duplicates page.
Following a review of the Duplicate Entities these can then be either;

• Duplicate is mergedFrom the File Duplicates page to a single Entity which inherits all attributesproperties, notes & tags, links,attachments, related documents and Files, summary. from the merged Entities.
• Duplicate Entity is correctedBy editing the Entity's properties by editing the properties where incorrect data has been added.
• Duplicate Entities are deemed not to actually be duplicates and can be removed from the duplicates list.

To access and manage Entity Duplicates, open the File menu and select 'Duplicates' from the File menu.
Any Potential Duplicates are listed in the left side pane.
The Duplicate Preview pane in the middle shows the Potential Duplicate Entity's properties.
The Existing Entity Preview pane on the right shows the existing Entity's properties. The existing Entity is the original/first Entity created that the Duplicate Entity has matched to.

View Potential Duplicate

Click on the Entity name\title of the potential duplicate from the list to preview both the potential duplicate's properties and the existing Entity's properties.

Search or Filter Duplicates

The Duplicates list can be searched by Entity Title\name and or filtered by Entity Type.

Search for a Potential Duplicate.

Enter the Entity's Title \ name part ofSearch is based on an exact match - e.g the Partial search function is not available in Duplicates search into the search field and either click the Search button or Enter from your keyboard.

Filter Duplicate list by Type

To filter the Potential Duplicate list by Entity Type, click onto the Entity Types filter field and select the required Type or Types and click OK. The list will automatically update.

Review Potential Duplicates

The first step is to review the potential Duplicate Entity and the Existing Entity's properties in the preview panes to identify the duplicate properties.
For a more in-depth review before merging, perform a general Search and review the The Entity Profile of both the Potential Duplicate and the Existing Entity.

If on review the Entity has been identified as a duplicate due to a data entry error - e.g 2 Entities with the same unique ID or a spelling mistake etc - this can be corrected by updating the Entity property or properties in the Entity profile. Once corrected this will remove the Entity from the potential duplicates.

If following the review it has been determined that the Entity duplicates are the same and should be merged then continue as follows to Merge Entities.

Warning: Once Entities have been merged, they cannot be unmerged so ensure to review first.

Merge Duplicate Entities

The first Entity at the top of the list auto populates the New Duplicate Preview pane and the Existing Entity Preview pane. To view another Entity and their Properties, from the list click on the Entity name/title to populate the Duplicate and the Existing Entity's properties.

1. Optional - Search the list for a specific Entity name, or filter the Entity list by Type\s.
2. From the list of potential duplicates click on the Entity name to preview* the Duplicate and the Existing Entity
   
3. To select an Entity to merge, in the left pane click into the check-box to the left of the Potential Duplicate Entity.
   
4. With the Potential Duplicate\s selected (checked), click 'Merge Checked Entities' at the top right. The Duplicate Entities are merged and removed from the Duplicates list.
   

Selecting multiple Entities for merging

It is best practice to merge Entities one at a time as the selected potential duplicate Entity's properties can be viewed alongside the Existing Entity that it will be merged to.
If multiple potential duplicate Entities are selected you need to click on each potential duplicate to review both the potential duplicate and the existing Entity it will be merged to, to ensure they are in fact the same Entity.

* Expand preview content to view details. Where content (of any of the available properties) is too long to show on the same line as the label - all properties are collapsed and can be viewed by clicking on the ellipsis alongside each property to view.

Or try closing the File and My Recents menus and\or use Ctrl + scroll out from your mouse or device.

Or via the browser control (accessed via the 3 dots at the top right of the browser) and reduce the content view by clicking the minus to make text smaller.

The Merged Entity

Once the Entities are merged to a single Entity, any subsequent editsRemembered term changes must be updated from the Entity within a related Document or added details update the Entity within every Document that merged Entity exists.

In the Entity Profile, properties with identical values are merged. Properties of the same type but with differing values are not merged so these properties are added to the Entity.
In the following example 3 properties of the same type had different values so all are included in the merged Entity. These can be 'tidied-up' by editing and \ or removing the properties within the Entity Profile or from the Entity record within a related Document.

Note: Potential Duplicate Entities are also identified and can be merged within the Add Entity process in Documents.

Where there are Entities on the duplicate Entities list that on review should not be merged, these can be removed from the File Duplicates list.

To remove Entities from the Potential Duplicates list;

1. Select the Entity\s by clicking into the checkbox to the left of the Duplicate from the potential Duplicates list
2. Next select 'Remove Checked Entities' from the top of the page and confirm.

This work-around can be used If Entities have been removed from the Duplicates list via the 'Remove Checked Entities' process and then subsequently determine that they should be merged. This can be achieved via the Document - 'Add Entity' process and may also require temporary existing Entity edits or even a new temporary Entity added.

This process will require some knowledge of the Entity Types Properties duplicate values.

1. Open any Document (in edit mode) from the same File as the Entities that were removed from the Duplicate list
2. Open the Add Entity window selecting the same Entity Type (e.g Person)
3. Search for just 1 property (e.g First name or Surname -not both) of the unmerged Entities. The removed Duplicate Entities will show and other Entities with the same property value. (If the Merge Duplicates window shows at this point you can jump ahead to step 10).
4. If the 'Merge Duplicates' message doesn't show then identify an existing Entity to edit and close the Add Entity window. You will need to take note of the duplicate Entity properties values to copy.
   
5. Search and open the profile of Entity identified and edit any property\s required to produce a Duplicate. Take note of the edits so they can be reinstated at the end of this process - (or you could create a new Entity to match the original Entities and delete after they
6. have been merged).
   
7. Repeat steps 1- 3 above which should now include the edited (or newly added) Entity. This will trigger the Duplicate being identified and the Merge Duplicate message shows.
8. Select Yes from the Merge Duplicates message.
   
9. From the Merge Checked Entities window - 1: Select the original Entity (only 1 of these will be enabled to check initially). 2: Select the other original Duplicate Entity which is now enabled. 3: Select 'Merge Checked Entities'.
   
10. If edits were made to an existing Entity now go back and re-edit the Entity (or delete the newly created Entity) used to match the original Entities to reinstate the original property values.